Deixis and Conjunction in Multimodal Systems
نویسنده
چکیده
In order to realize their full potential, multimodal interfaces need to support not just input from multiple modes, but single commands optimally distributed across the available input modes. A multimodal language processing architecture is needed to integrate semantic content from the different modes. Johnston 1998a proposes a modular approach to multimodal language processing in which spoken language parsing is completed before multimodal parsing. In this paper, I will demonstrate the difficulties this approach faces as the spoken language parsing component is expanded to provide a compositional analysis of deictic expressions. I propose an alternative architecture in which spoken and multimodal parsing are tightly interleaved. This architecture greatly simplifies the spoken language parsing grammar and enables predictive information from spoken language parsing to drive the application of multimodal parsing and gesture combination rules. I also propose a treatment of deictic numeral expressions that supports the broad range of pen gesture combinations that can be used to refer to collections of objects in the interface.
منابع مشابه
Interface Features Affecting Deixis Production: a Simulation Study
This paper examines gestural deixis production in a multimodal (natural language and pointing) interaction. Users (N=30) were required to fill in 4 forms by interacting with a simulated system. Labelling of Entry Fields (LEF, Complete-Partial; between-subject) and Feedback (Present-Absent within-subject) were manipulated in a 2*2 design. Results indicate that deixis occurred more frequently in ...
متن کاملIncremental Generation of Multimodal Deixis Referring to Objects
This paper describes an approach for the generation of multimodal deixis to be uttered by an anthropomorphic agent in virtual reality. The proposed algorithm integrates pointing and definite description. Doing so, the context-dependent discriminatory power of the gesture determines the contentselection for the verbal constituent. The concept of a pointing cone is used to model the region single...
متن کاملIntegration of Speech and Deictic Gesture in a Multimodal Grammar
In this paper we present a constraint-based analysis of the form-meaning mapping of deictic gesture and its synchronous speech signal. Based on an empirical study of multimodal corpora, we capture generalisations about well-formed multimodal utterances that support the preferred interpretations in the final context-of-use. More precisely, we articulate a multimodal grammar whose construction ru...
متن کاملInteraction of Speech, Deixis and Graphical Interface
To solve certain problems of multimodal interaction the concept of graphical utterances is introduced. Two different functions of deictic gestures are discussed: deictic gestures may be used to focus on a certain context of interpretation and they may be used to provide a referent for an natural language expression. The relations between deictic gestures and visual utterances are presented. Pro...
متن کامل